from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(against_lib="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	2.345982	0.197174	NaN	...	brute	-1	1	0.663	0.351551	0.008051	1.000	6.673228	6.674977	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.027585	0.003773	NaN	...	brute	-1	1	1.000	18.881258	0.082581	0.757	0.001461	0.001461	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	3.136109	0.100838	NaN	...	brute	-1	5	0.757	18.757458	0.110708	0.882	0.167193	0.167196	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.443875	0.027407	NaN	...	brute	1	100	0.882	0.344548	0.008601	1.000	7.092985	7.095194	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.025788	0.002638	NaN	...	brute	1	100	1.000	18.873383	0.101548	0.757	0.001366	0.001366	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	3.139071	0.062996	NaN	...	brute	-1	100	0.882	19.419744	0.041208	0.663	0.161643	0.161644	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.442894	0.022442	NaN	...	brute	1	5	0.757	0.284448	0.009618	1.000	8.588180	8.593088	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.026557	0.001892	NaN	...	brute	1	5	1.000	4.268789	0.081962	0.922	0.006221	0.006222	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.621578	0.010942	NaN	...	brute	1	1	0.663	4.254634	0.047411	0.929	0.381132	0.381156	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.851052	0.028637	NaN	...	brute	-1	1	0.896	0.270214	0.008247	1.000	6.850320	6.853510	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.005521	0.001625	NaN	...	brute	-1	1	1.000	4.374117	0.123455	0.922	0.001262	0.001263	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.809051	0.070351	NaN	...	brute	-1	5	0.922	4.355370	0.067786	0.896	0.644963	0.645041	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.001	5.744	0.0	-1	1	19.253	0.126	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.001	5.935	0.0	-1	5	0.354	0.014	1.000	0.038	0.038	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.000	5.890	0.0	1	100	18.638	0.238	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.001	5.652	0.0	-1	100	0.366	0.021	1.000	0.039	0.039	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.014	0.001	5.914	0.0	1	5	4.372	0.123	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.015	0.001	5.358	0.0	1	1	0.268	0.008	1.000	0.056	0.056	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.314	0.0	-1	1	4.262	0.064	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.005	0.000	0.326	0.0	-1	5	0.278	0.005	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.346	0.197	0.0	0.002	-1	1	0.352	0.008	1.000	6.673	6.675	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.028	0.004	0.0	0.028	-1	1	18.881	0.083	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.136	0.101	0.0	0.003	-1	5	18.757	0.111	0.882	0.167	0.167	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.003	0.0	0.027	-1	5	0.352	0.011	1.000	0.077	0.078	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.444	0.027	0.0	0.002	1	100	0.345	0.009	1.000	7.093	7.095	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.003	0.0	0.026	1	100	18.873	0.102	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.139	0.063	0.0	0.003	-1	100	19.420	0.041	0.663	0.162	0.162	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.029	0.003	0.0	0.029	-1	100	0.363	0.009	1.000	0.079	0.079	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.443	0.022	0.0	0.002	1	5	0.284	0.010	1.000	8.588	8.593	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.002	0.0	0.027	1	5	4.269	0.082	0.922	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.622	0.011	0.0	0.002	1	1	4.255	0.047	0.929	0.381	0.381	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.001	0.0	0.024	1	1	0.267	0.008	1.000	0.091	0.091	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.851	0.029	0.0	0.002	-1	1	0.270	0.008	1.000	6.850	6.854	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.002	0.0	0.006	-1	1	4.374	0.123	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.809	0.070	0.0	0.003	-1	5	4.355	0.068	0.896	0.645	0.645	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.007	0.002	0.0	0.007	-1	5	0.269	0.008	1.000	0.026	0.026	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.920315	1.186926	NaN	...	kd_tree	-1	1	0.929	2.917273	0.272286	1.000	0.315471	0.316842	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.003030	0.000589	NaN	...	kd_tree	-1	1	1.000	143.781297	0.000000	0.946	0.000021	0.000021	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	1.125627	0.559873	NaN	...	kd_tree	-1	5	0.946	143.639217	0.000000	0.951	0.007836	0.007836	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	6.058615	0.722357	NaN	...	kd_tree	1	100	0.951	3.033202	0.199385	1.000	1.997432	2.001743	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003938	0.001777	NaN	...	kd_tree	1	100	1.000	145.110898	0.000000	0.946	0.000027	0.000027	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	3.511473	0.341730	NaN	...	kd_tree	-1	100	0.951	142.911426	0.000000	0.929	0.024571	0.024571	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.879582	0.409067	NaN	...	kd_tree	1	5	0.946	0.006562	0.000285	1.000	286.413569	286.683871	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001771	0.000448	NaN	...	kd_tree	1	5	1.000	0.047423	0.002399	0.911	0.037341	0.037389	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	1.018047	0.382193	NaN	...	kd_tree	1	1	0.929	0.069796	0.003577	0.894	14.585944	14.605084	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.038381	0.016261	NaN	...	kd_tree	-1	1	0.891	0.006373	0.000641	1.000	6.022782	6.053193	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.004434	0.001514	NaN	...	kd_tree	-1	1	1.000	0.049109	0.005837	0.911	0.090287	0.090922	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.034384	0.001957	NaN	...	kd_tree	-1	5	0.911	0.044800	0.002006	0.891	0.767518	0.768287	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	2.844	0.034	0.028	0.0	-1	1	141.107	0.000	0.929	0.020	0.020	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.907	0.115	0.020	0.0	-1	5	3.098	0.290	1.000	1.261	1.267	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.972	0.119	0.020	0.0	1	100	148.515	0.000	0.951	0.027	0.027	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.920	0.089	0.020	0.0	-1	100	3.108	0.274	1.000	1.261	1.266	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.037	0.095	0.020	0.0	1	5	0.046	0.002	0.891	87.903	87.967	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.973	0.100	0.020	0.0	1	1	0.006	0.000	1.000	687.569	688.424	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.016	0.0	-1	1	0.069	0.002	0.894	0.015	0.015	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.021	0.0	-1	5	0.006	0.000	1.000	0.123	0.123	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.920	1.187	0.0	0.001	-1	1	2.917	0.272	1.000	0.315	0.317	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.0	0.003	-1	1	143.781	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.126	0.560	0.0	0.001	-1	5	143.639	0.000	0.951	0.008	0.008	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.000	0.0	0.004	-1	5	2.933	0.231	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	6.059	0.722	0.0	0.006	1	100	3.033	0.199	1.000	1.997	2.002	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.002	0.0	0.004	1	100	145.111	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.511	0.342	0.0	0.004	-1	100	142.911	0.000	0.929	0.025	0.025	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.006	0.001	0.0	0.006	-1	100	3.129	0.267	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.880	0.409	0.0	0.002	1	5	0.007	0.000	1.000	286.414	286.684	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.0	0.002	1	5	0.047	0.002	0.911	0.037	0.037	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.018	0.382	0.0	0.001	1	1	0.070	0.004	0.894	14.586	14.605	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.0	0.001	1	1	0.006	0.001	1.000	0.198	0.199	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.038	0.016	0.0	0.000	-1	1	0.006	0.001	1.000	6.023	6.053	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.004	0.002	0.0	0.004	-1	1	0.049	0.006	0.911	0.090	0.091	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.034	0.002	0.0	0.000	-1	5	0.045	0.002	0.891	0.768	0.768	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.0	0.003	-1	5	0.006	0.001	1.000	0.432	0.435	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.193181	0.004653	300	...	100	100	10	0.824	0.494395	0.008458	1.0	0.390743	0.3908	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	128.674	0.0	300	0.001	0.001	0.6	0.058	0.824	214.629	215.625	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.193	0.005	300	0.004	0.0	0.494	0.008	1.0	0.391	0.391	See	See